A nnlirlnrr o Waoro 

"F FV tu 6 


V T VU1 «* 


bje Voice-Activated Computer to Instructional Applications in 
Clean Room Environments 


C. A. Graves, Electrical and Computer Engineering Department 
North Carolina A & T State University, Greensboro, NC 
Mark L. Lupisella, Real-time Software Engineering Branch, 
NASA Goddard Space Flight Center, Greenbelt MD 


ABSTRACT 

The use of wearable computing technology in restrictive 
environments related to space applications offers promise 
in a number of domains. The clean room environment is 
one such domain in which hands-free, heads-up, wearable 
computing is particularly attractive for education and 
training because of the nature of clean room work. We 
have developed and tested a Wearable Voice- Activated 
Computing (WEVAC) system based on clean room 
applications. Results of this initial proof-of-concept work 
indicate that there is a strong potential for WEVAC to 
enhance clean room activities. 
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1. Introduction 

Over the past two decades, wearable computing 
technology has been applied in several areas including 
medicine, the military, industry, law enforcement, 
security, and education (particularly for the disabled and 
workers in restrictive environments) [4,5]. At Goddard 
Space Flight Center (GSFC), efforts in applying wearable 
computing to the NASA mission began in 2000, with the 
initiation of the WEVAC project in the Real-time 
Software Engineering Branch (Code 584) [2]. Three 
work environments relevant to NASA missions are being 
considered for use of WEVAC: (1) clean room, (2) launch 
pad, and (3) human space flight. Investigating clean room 
environments is a near-term priority because they are the 
most accessible of the three. This work will eventually 
expand research to include the launch pad and space 
environments. In this particular effort we seek to use the 
WEVAC system for speech recognition-driven 
multimedia in “embedded” instruction of tasks that 
require a high degree of hand-eye (or body-eye) 
coordination. Simply put, a person would be able to leam 
while doing, rather than viewing education material 
related to learning a physical task ahead of time, taking 
notes, and then referring to those notes while trying to 


perform the task. Such an “embedded” learner scenario 
has been studied extensively by the authors in [1,2]. Two 
types of “embedded” instruction are of interest: (1) self- 
guided instruction and (2) remote instruction. 

The goal of this research was to use the current 
WEVAC system to evaluate the practicality of self-guided 
instruction and remote instruction within the clean room 
environment Efforts were also made to extend WEVAC 
functionality for instruction enhancement in an academic 
microelectronics clean room environment. Aside from the 
clean room being the most accessible of the three targeted 
environments, four other factors help to make a 
compelling case for applying wearable computing in this 
environment: (1) step-by-step procedures often need to be 
followed in the clean room, (2) tasks in a clean room 
usually require the hands free operation and coordination 
between the eyes and the rest of the body, (3) accessing 
and collecting data in many forms is required in the clean 
room, and (4) a computer wom under a clean room 
“bunny” suit (clean room suit) will cause less of a 
contamination concern than a laptop carried around 
outside of the bunny suit. The authors are not aware of 
any other studies geared toward using wearable 
computing in a clean room environment. 

The remainder of this paper is organized as 
follows. Section II describes the state of the current 
WEVAC system. The three experiments conducted 
involving self-guided and remote instruction using 
common commercial software, are discussed in Section 
III. Finally, a summary of the work conducted is given in 
Section IV along with a perspective on possible future 
work. 


2. Present WEVAC System 

The following subsections describe the components of the 
current WEVAC system. This system was used to 
conduct all experiments subsequently discussed. 

2.1 Wearable PC 

Figure 1 depicts the actual wearable PC used, which is the 
Xybemaut MA V (Mobile Assistant 5). This unit is the 
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heart of the wearable system. It is designed to be belt- 
attachable, has dimensions of 5.9” x 3.5” x 2”, and weighs 
approximately 1 lb. It is equipped with a 500 MHz Intel 
Mobile Celeron processor, 128 MB of RAM, and a 5 GB 
hard drive. Standard ports for interfacing include USB, 
FireWire, and Compact Flash. 



Figure 1: Xybernaut MA V Wearable PC 


2.2 WE VAC Input Device 

The WEVAC system allows four different modes of 
input. Two of the input modes require the use of at least 
one hand, and the other two modes are completely hands- 
free. As shown in Figures 2(a) and 2(b), the Off-Table 
Track Mouse and the Wrist-Worn mini-keyhoard 
constitute the one-handed input devices. Figures 2(c) and 
2(d) show the hands-free input devices, which are the 
Plantronics digital audio headset (microphone) and the 
IBOTpro web camera. All of these input devices interface 
with the MA V through the USB port except for the 
camera, which goes through the FireWire port. 


Xybernaut Flat Panel Display (FPD) and the 
MicroOptical clip-on viewer (Head Mounted Display), 
shown in Figures 3(a) and 3(b). The small FPD is 
typically attached to a user such that it can be accessed on 
an as-needed basis (usually for application setup 
purposes). The MicroOptical clip-on display is mtended 
to be the primary source of video output for the system 
since it allows heads-up viewing. The clip-on display 
outputs VGA resolution video and covers only one of the 
user’s eyes (monocular display). Thus, when the clip-on 
display is viewed with both eyes open, a see-through 
effect is perceived. The earphones of the Plantronics 
audio headset are used for audio output. 




(a) 


<b) 



(c) 

Figure 3: WEVAC Output Devices (a) Xybernaut 
Xyber Panel Flat Panel Dispaly, (b) MicroOptical 
SV-9 PC Viewer, (c) Plantroincs Digital Audio Head 
Set 



Figure 2: WEVAC Input Devices (a) Off-Table 
Track Mouse, (b) Xybernaut Wrist-Worn Mini- 
Keyboard, (c) Plantronics Digital Audio Headset, 
(d) iBOTpro Web Camera 


2.3 WEV AC Output Devices 

Like most traditional computer systems, the primary 
forms of user output provided by WEVAC are video and 
audio. Two video output devices are used. These are the 


2.4 WEVAC Wireless Communication 

An essential feature for WEVAC is the ability to send and 
receive data via a wireless network. Networking is needed 
for sharing multimedia data with other computers as well 
as for general access to the Internet. Wireless 
communication is needed because the users of WEVAC 
are expected to be highly mobile and actively performing 
physical tasks. This means that they cannot afford to be 
restricted by network cabling. The current system is able 
to interface with 802.1 lb-based wireless local area 
networks (WLANs). The WLAN card currently used is 
manufactured by Orinoco, and inserted into the wearable 
computer in Figure 1 . 

2.5 WEVAC Specific Software 

The WEVAC system currently is running the Windows 
2000 operating system. Although there are many 
programs stored on the computer, those specifically used 
in initial experiments presented here include (1) Dragon 
NaturallySpeaking for speech recognition, (2) 
Macromedia Authorware Web Player for slide show 
presentations, (3) Microsoft Windows Media Player for 
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video playback, and (4) Microsoft Windows NetMeeting 
for multimedia sharing. 

2.6 System Integration 

All hardware components of the WEVAC system and the 
appropriate interconnecting cables are attached to a vest 
which the user can put on and take off conveniently, 
without having to attach or remove each wearable 
component to or from the body each time the system is 
used. The integrative vest is shown in Figure 4. 



Figure 4: Vest to Integrate the Wearable Components 

3. Experiments Conducted 

With the hardware and software described in the previous 
section, three voice-driven experiments were set up. Two 
of these experiments involved self-guided instruction and 
the third involved remote instruction. Self-guided 
instruction experiments included one that utilized static 
multimedia (images and text) and one that utilized 
dynamic multimedia (video and audio). 

3.1 Experiment #1: Static Multimedia for Self- 
Guided Instruction 

Experimental Setup 

The first experiment involved accessing static multimedia 
for self-guided instruction purposes. The term “static 
multimedia instruction” is used to refer to a set of 
instructions given as a collection of text and images. A 
traditional slide presentation is an example of this. This 
experiment was developed based on a NASA GSFC 
online tutorial showing how to put on a bunny suit prior 
to entering the clean room — which was intended 
primarily to demonstrate the concept of enhanced 
training. This tutorial was generated as a -web-based slide 
presentation using the Authorware Web Player by 
Macromedia. Each slide in the presentation represents a 
distinct step in the process of putting on the bunny suit. 
Two non-consecutive example slides from the bunny suit 
tutorial are shown in Figure 5. The slide in Figure 5(a) 
shows a step early in the process in which the garments 
are retrieved. The slide in Figure 5(b) represents a step 


sometime later in the process where the user is about to 
put on the gown. Note the corresponding text at the 
bottom of each slide. 



(a) (b) 


Figure 5: Static Multimedia Example 

Now imagine that the person in Figure 5 is 
equipped with an internet-connected wearable computing 
system such as WEVAC and can view this tutorial 
presentation while he is actually in the process of 
gowning. Alternatives to this would be to either (1) view 
this tutorial online some time prior to entering the clean 
room gowning area while taking notes (mental or 
written), and then taking these notes into the clean room 
gowning area to refer to while putting on the bunny suit, 
or (2) carrying an internet-connected laptop computer into 
the gowning area and referring to the presentation in that 
way. We believe that the wearable approach would be a 
more effective and efficient option for learning the bunny 
suit gowning procedure (given a comfortably attached 
wearable system) because it would cause minimal 
distraction to the user, as well as requiring the user to 
remember fewer steps. The concept for enhancing this 
simple gowning procedure using wearable computing and 
static multimedia can be applied to more complicated 
clean room procedural tasks. 

Vocabulary for Static Multimedia Instruction 

In the WEVAC system, the above tutorial was setup to be 
viewed via voice commands. In this and all other 
experiments, the user always has the ability to use the 
built-in Windows navigation vocabulary in the Dragon 
software (i.e., mouse manipulation, window swapping, 
etc.). Additionally, each experiment required the creation 
of a custom vocabulary. For the case of static multimedia 
access, only two custom vocabulary words were created. 
These words are shown below, along with a description of 
the action taken when the words are recognized. 

1 . Next - Progress to the next slide. 

2. Previous - Go back to the previous slide 

WEVAC Custom Command Protocol 

Each custom WEVAC command in this and other 
experiments follow the same activation protocol. Prior to 
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the initiation of any custom WEVAC voice command, we 
assume that the system is in the Asleep state, where the 
only word that will be recognized is the “WEVAC” wake 
up command. The protocol is as follows: 

1. From the Asleep state, say “WEVAC” to get the 
system’s attention. 

2. Once the beep sound is heard (indicating that you have 
the systems attention), then say the custom WEVAC 
command (e.g. “next”). 

3. Specialized computer instructions will execute based 
on the custom command recognized. 

4. The system goes back into the Asleep state and beeps 
as an indication of this. 

The Dragon NaturallySpeaking scripting language is used 
for voice command programming. Programs are stored in 
Dragon voice command files (with the .dvc extension). 

Results of Experiment #1 

For this experiment there is a very small custom 
vocabulary and each of the commands lead to a simple 
keystroke sequence being sent to the system. Thus, no 
problems arose related to recognizing and executing the 
custom commands. The only significant limitation 
observed while conducting Experiment #1, was that the 
Authorware output didn’t fit completely on the VGA 
screen of the WEVAC system. One solution to this 
problem would be to utilize a higher resolution wearable 
display. Drawbacks of this solution would be bulkier (and 
heavier) display electronics, as well as a large, more 
obstructive computer screen, floating in front of the user. 
A more feasible solution to the problem would be to 
create the static multimedia presentation with a software 
tool that allows the size of the presentation slides to scale 
according to the size of the application window (e.g. MS 
PowerPoint). 

3.2 Experiment #2: Dynamic Multimedia for Self- 
guided Instruction 

Experimental Setup 

“Dynamic multimedia instruction” refers to instructions 
given in video form (with accompanying audio). 
Additionally, static multimedia may be included as part of 
dynamic multimedia presentations. For example, it may 
be useful to overlay a segment of instructional video with 
bulleted text information. Also, most traditional 
presentation tools, such as PowerPoint, are capable of 
including video and audio clips within a presentation. For 
this experiment, we used Windows Media Player as the 
instruction delivery tool. We used professionally 
produced clean room instructional MPEG encoded videos 
that pertain specifically to microelectronics clean rooms. 
These videos can serve as a model for other institutions 
and organizations that want to produce such instructional 
material. Again, we feel that it would be beneficial for 


clean room personnel to view these instructional videos 
using a wearable computer while in the actual 
environment working on specialized equipment. 

It is important to note that an intermediate 
process needs to be performed before the videos can be 
used effectively in the WEVAC system. This process 
involves segmenting a long video clip into many small 
clips and is essential for allowing easier voice-activated 
navigation because the single stream of video (for a 
particular tutorial) is segmented into clips representing 
distinct steps in the process. For example, in this 
experiment an equipment tutorial video was segmented 
into 24 separate video clips using a video-editing tool. 
Jumping between major steps in the equipment tutorial 
only requires skipping from one clip to another (either 
forward or backward), rather than fast-forwarding or 
rewinding and visually trying to determine when to stop 
at boundaries separating major steps of the procedure. 
The user, however, is still allowed to use fast forward, 
rewind, and stop commands within a video clip (major 
step) if needed. 

Vocabulary for Dynamic Multimedia Instruction 

Again, the built-in vocabulary for Dragon Naturally 
speaking is used. The customized WEVAC vocabulary 
for this experiment is given below. 

1. Proceed - Play video clip from current 
position. 

2. Freeze - Pause clip at the current position. 

3. Full Screen - Display full-screen version of 
the current video clip being played. 

4. Fast Forward - Move forward through the 
current video clip at a high speed. 

5. Rewind - Move backward through the current 
video clip at a high speed. 

6. Next - Jump to the start of the next video clip 
(next step) and begin playing. 

7. Previous - Jump to the start of the previous 
video clip (previous step) and begin playing. 

8. Repeat - Return to the beginning of current 
video clip (current step) and begin playing. 

9. Stop It - Return the clip position to the 
beginning of the current clip (current step) and 
stop playing the video. 

10. Volume Up - Adjust volume upward by one 
increment. 

11. Volume Down - Adjust volume downward 
by one increment. 

12. Volume Mute - Toggle between muted and 
non-muted audio. 

Results of Experiment #2 

The cited limitations and associated recommendations 
resulting from conducting dynamic multimedia, self- 
guided instruction with WEVAC are compiled in Table 1. 
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Where feasible, the recommendations were implemented 
on the current WEVAC system, as indicated in the table. 


Table 1: Summary of Experiment #2 Results 


Limitations 

Recommendations 

Some voice commands 
associated with long keystroke 
sequences are not always 
executed properly on first 
attempt. 

Inserting “waif’ commands between 
individual commands in the voice 
command script helps. 

Repeating the commands several 
times works. 

The video output from the 
Media Player is not seen in the 
wearable Head Mounted 
Display 

Reducing the graphic acceleration 
setting to about half of die maximum 
setting solves this problem. 

Speech recognition software 
performance is greatly reduced 
by playing video at the same 
time. 

Obtaining a higher performance 
wearable computer will improve 
poor performance. 


3 3 Multimedia Sharing for Remote Instruction 

Experimental Setup 

The final experiment was based on the idea of two users, 
an instructor and a student, being able to share multimedia 
information through a wireless network. The instructor is 
envisioned as being stationary in her office during the 
instruction session, while the student is moving freely in a 
lab-like learning environment, such as a clean room, 
while wearing a wearable computer such as WEVAC. 
The idea is that the instructor and the student may not be 
able to occupy the learning environment at the same time. 
For example, the instructor may need to be in her office to 
conduct office hours, while at the same time one of her 
graduate students needs assistance from her to perform a 
very specialized task in the laboratory. In the case of 
clean room instruction, an instructor may not find it to be 
useful or efficient to “suit up” in the bunny suit and enter 
the clean room to help with a minor task. 

In this experiment, “sharing of multimedia data” 
refers to each party (the instructor and the student) being 
able to observe and control the multimedia information of 
concern for instructional purposes. Windows NetMeeting 
fit our needs for this experiment. It was originally 
intended for teleconferencing purposes over long 
distances. The key feature of NetMeeting that is useful for 
remote instruction is the desktop sharing feature, which 
allows either of the users to observe and control the other 
user’s desktop upon request. 

A student working alone in a lab may require the 
assistance of a remote instructor. WEVAC should allow 
the student to initiate a remote instruction session and 
communicate verbally and visually with the instructor. In 
many cases it may be useful for the student to transmit 
video (using the wearable camera) to his own desktop, 
and then share his desktop with the instructor. In this way, 
the instructor and the student will not only be able to see 
the same view, but if needed, the instructor can take 
control of the student’s cursor in order to point out 


specific items of interest in the lab environment in which 
the instructor is not located. Note that this works ideally 
when the person sharing their desktop is operating with a 
lower display resolution than the other person. The 
person with the wearable system (student) will likely be 
working with a lower resolution display than the other 
person (instructor) who will likely be working with very 
high resolutions offered by modem desktop and laptop 
machines. In the opposite case where the instructor would 
be sharing her desktop with the student, her higher 
resolution screen would not show up completely on the 
student’s lower resolution screen. Thus, the student would 
have to do some window scrolling in order to access the 
entire content of the instructor’s desktop. This would be 
inconvenient, but not prohibitive. Fortunately, the first 
situation of a student sharing his desktop is the more 
likely scenario. A low resolution display is a short- 
coming of wearable computer systems, but the technology 
for these displays is evolving. 

Vocabulary for Remote Instruction 

Below is the custom vocabulary developed for 
Experiment #3. 

1 . Send Video - Start transmitting video frames 
from the WEVAC camera. 

2. Snap Shot - Hold the current video frame. 

3. Hang Up - Disconnect from current remote 
session with instructor. 

4. Single Size - Return the selected video 
w'indow to its original size. 

5. Double Size — Make the selected video 
window twice its original size. 

6. Triple Size - Make the selected video window- 
three times its original size. 

7. Quadruple Size - Make the selected video 
window- four times its original size. 

8. Share Desktop - Share WEVAC desktop 
with remotely connected user and automatically 
allow control of desktop when requested. 

9. Accept Calls - Accept incoming calls 
automatically. 

10. Unshare - Disable sharing of the WEVAC 
desktop. 

11. Do Not Disturb - Prevent incoming calls 
from being seen or heard. 

Results of Experiment #3 

The cited limitations and associated recommendations 
resulting from experimenting with remote instruction 
using WEVAC are shown in Table 2. Again, w-here 
feasible, the recommendations were implemented on the 
current WEVAC system. 
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Table 2: Summary of Experiment #3 Results 


Limitations 

Recommendations 

With a single microphone, only 
the speech recognition or the 
remote audio communication can 
be implemented at any given 
time, but not both. 

Using two microphones, one for 
speech n^o^sUtiOE and the other for 
remote audio communications 
corrects this. (USB technology 
makes this possible) 

USB-based video and USB- 
based audio connected 
simultaneously cause overload 
on USB resources 

Replacing the USB-based camera 
with a FireWire-based camera 
corrects this. 

Communication lag for all types 
of data. 

Ensure high bandwidth network 
availability. 


4. Conclusions 


In summary, we believe the WEVAC system is useful in 
clean room environments for the following reasons: 

1) The clean room is a manual work-intensive 
environment that could benefit from hands-free 
computer use. 

2) Higher fidelity training is possible because the actual 
environment is used for training, leading to better 
overall performance of tasks. 

3) More efficient training is possible, which requires 
less time, less people, and less paper documentation. 

4) Contamination of the clean room environment with 
wearable computer systems is less of a concern than 
with traditional systems because the wearable 
computer can potentially be concealed within the 
bunny suit. 

We were successful in demonstrating the possibility of 
using a wearable computer system for instructional 
enhancement in both the NASA and academic clean room 
environments. Self-guided and remote instruction 
experiments were conducted using NASA’s WEVAC 
system. At the time of these experiments, commercial 
wearable computing devices were still too bulky to be 
concealed within a clean room bunny suit comfortably. 
Thus, a usability analysis in practical clean room 
situations was not feasible. Our experiments with the 
WEVAC system lead us to recommend that wireless 
technology be used as much as possible for 
interconnection between the wearable computer and 
peripheral devices in the future. This is due to the 
problems encountered when trying to conceal the system 
cables while attaching devices to the integrative vest. The 
next generation of WEVAC is expected to take advantage 
of the latest advances in computing and communication 
technology. The next WEVAC system can expect to have 
a smaller more powerful CPU, with many of the 
input/output devices connected wirelessly by way of 
Bluetooth or Ultra Wideband (UWB) technology'. 

Future research will compare the performance of 
a fully wearable system (like the current WEVAC system) 
to that of a system where only the I/O is wearable. In a 
wearable I/O only system the Head Mounted Display, 


audio headset, mouse and keyboard will be interfaced to a 
non-wearable computer via wireless connections. 
Another wearable computer system variation to be 
examined is one in which a lighter-weight, less power- 
hungry Palm PC-type device (perhaps Windows CE- 
based) could be incorporated as a wearable, wireless, 
remote terminal for a high-performance desktop machine. 
It will be interesting to assess the pros and cons of these 
three systems across a wide range of educational 
enhancement related applications. Factors that will be of 
interest in the evaluations would include (but not be 
limited to) power consumption, heat dissipation, I/O and 
computational latency, bulkiness, and reliability [4,5]. 
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